SPECULATIVE DECODING

TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs (2026.findings-eacl)

Challenge:	Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs.
Approach:	They propose a model that uses a constant-sized key-value cache to train long-context models.
Outcome:	Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks.

Copied to clipboard

Minjae Lee, Wonjun Kang, Byeongkeun Ahn, Christian Classen, Kevin Galim, Seunghyuk Oh, Minghao Yan, Hyung Il Koo, Kangwook Lee

Robustness in NLP

Challenge:	Large Vision Language Models (LVLMs) are advanced models that process multiple modalities, such as images, audio, and video, alongside text.
Approach:	They propose to use a method to generate and verify draft tokens in parallel . they compare existing methods with small draft models and observe performance fluctuations .
Outcome:	The proposed method achieves an average walltime speedup of 1.74 over autoregressive decoding and a 5% improvement over single drafting methods.

Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding (2025.findings-naacl)

Copied to clipboard

Sukmin Cho, Sangjin Choi, Taeho Hwang, Jeongyeon Seo, Soyeong Jeong, Huije Lee, Hoyun Song, Jong C. Park, Youngjin Kwon

Benchmarking the Energy Savings with Speculative Decoding Strategies (2026.findings-eacl)

Challenge:	Existing methods for drafting and verifying tokens require significant fine-tuning or have inconsistent performance across tasks.
Approach:	They propose a lossless drafting approach that organizes various token sources into multiple databases in a hierarchical framework based on temporal locality.
Outcome:	The proposed method outperforms existing database drafting methods on Spec-Bench using 7B and 13B parameters.

Copied to clipboard

Rohit Dutta, Paramita Koley, Soham Poddar, Janardan Misra, Sanjay Podder, Naveen Balani, Saptarshi Ghosh, Niloy Ganguly

Challenge:	Existing studies on speculative decoding have focused on the energy requirements of these models, despite their utility and utility.
Approach:	They propose to analyze the energy requirements of speculative decoding strategies and analyze how various factors influence the energy optimizations.
Outcome:	The proposed approach reduces decoding time while offloading a substantial portion of the sequential generation to a smaller, more efficient model.

An Empirical Study of Speculative Decoding for Small Language Models (2026.eacl-long)

Copied to clipboard

Luca Mainardi, Selcuk Sandikci, Joaquin Vanschoren

CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter (2025.acl-long)

Challenge:	Existing studies focus on 7B-70B parameters models, leaving a knowledge gap for small language models.
Approach:	They propose a draft-then-verify paradigm that allows for a single forward pass through a model and transfer of all model parameters to the GPU cache.
Outcome:	The proposed method can be used to accelerate small language models with low computational overhead.

Copied to clipboard

Yepeng Weng, Dianwen Mei, Huishi Qiu, Xujie Chen, Li Liu, Jiang Tian, Zhongchao Shi

UniSpec: Training-Free Speculative Decoding for Robust LLM Acceleration Across Languages and Hardware (2026.acl-long)

Challenge:	Existing methods that focus on training and inference suffer from misalignment . speculative decoding is a powerful technique that accelerates large language models .
Approach:	They propose a framework that improves both accuracy and efficiency in speculative drafting by using cross-step representational alignment.
Outcome:	The proposed framework outperforms existing methods on three LLM families and three benchmark datasets.

Copied to clipboard

Truong Dinh Do, Nguyen-Khang Le, Le-Minh Nguyen

Robustness in NLP

Challenge:	Existing methods for speculative decoding ignore device-specific verification costs and lack of mechanisms to assess draft token quality.
Approach:	They propose a training-free, lossless speculative decoding framework that enables robust, plug-and-play LLM acceleration across diverse hardware configurations and languages.
Outcome:	The proposed framework outperforms existing training-free methods while maintaining identical output quality across different hardware environments.

RASD: Retrieval-Augmented Speculative Decoding (2025.findings-acl)

Copied to clipboard

Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, Hao Henry Wang

Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling (2025.acl-long)

Challenge:	Existing methods for generating draft tokens rely on lightweight draft models or additional model structures to generate tokens and retrieve context from databases.
Approach:	They propose to use a pruning method to enhance model-based speculative decoding by combining the best-fit model with the best retrieval tree.
Outcome:	The proposed method achieves state-of-the-art inference acceleration across tasks such as DocQA, Summary, Code, and In-Domain QA.

Copied to clipboard

Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu

HCSpec: Two-Tier Horizontal Cascade Speculative Decoding for High-Efficiency Large Language Model Inference (2026.acl-long)

Challenge:	Large Language Models (LLMs) generate only one token at each decoding step, leading to high latency.
Approach:	They propose a speculative decoding paradigm that stores tokens in an adjacency matrix and employs a breadth-first-search algorithm to construct a draft tree.
Outcome:	The proposed method outperforms existing train-free methods by 30% and even a training method by 25%.

Copied to clipboard

Yizhou Zhang, Siming Chen, Hao Ye, Erhu Feng

EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025.naacl-long)

Challenge:	Existing approaches to decode large language models adopt a homogeneous architecture . autoregressive decoding is a bottleneck because tokens must be generated sequentially .
Approach:	They propose a framework that organizes heterogeneous position-specialized draft modules into a horizontal cascade.
Outcome:	The proposed framework outperforms the current state-of-the-art (EAGLE3) and achieves 3.72x acceleration over vanilla decoding.

Copied to clipboard

Yunsheng Ni, Chuanjian Liu, Yehui Tang, Kai Han, Yunhe Wang

Jakiro: Boosting Speculative Decoding via Decoupled MoE (2026.acl-long)

Challenge:	Speculative decoding is a key technique for enhancing the inference speed of Large Language Models.
Approach:	They propose a method that adds padding tokens to ensure that the number of new tokens remains consistent across samples.
Outcome:	The proposed method can handle the issue of inconsistent prediction tokens without adding padding tokens.

Copied to clipboard

Haiduo Huang, Fuwei Yang, Zhenhua Liu, Pengju Ren

BASS: Batched Attention-optimized Speculative Sampling (2024.findings-acl)

Challenge:	Existing methods to accelerate large language model inference have a fundamental limitation: candidates at the same tree layer share identical feature representations, constraining diversity and diminishing overall effectiveness.
Approach:	They propose a decoupled mixture of experts (MoE) into a draft model to generate diverse tokens from distinct feature spaces.
Outcome:	The proposed approach achieves significant speedups over strong baselines, with notable improvements in non-greedy scenarios where token diversity is crucial.

Copied to clipboard

Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, Anoop Deoras

HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding (2026.acl-long)

Challenge:	Speculative decoding has emerged as a powerful method to improve latency and throughput in hosting large language models.
Approach:	They propose a batched speculative decoding system that generates sequences at an average speed of 5.8ms per token and a batch size of 8 at a 2.15 speed-up over optimized regular decoding.
Outcome:	The proposed system achieves state-of-the-art latency and speed-up over optimized regular decoding.

Copied to clipboard

Siran Liu, Yang Ye, Qianchao Zhu, Zane Cao, Yongchao He

SAM Decoding: Speculative Decoding via Suffix Automaton (2025.acl-long)

Challenge:	Autoregressive decoding limits the inference throughput of Large Language Models due to its sequential dependency.
Approach:	They propose a framework that allocates verification effort in proportion to candidate uncertainty.
Outcome:	Speculative decoding achieves an average speedup over state-of-the-art methods . a small subset of high-confidence predictions accounts for most successful verifications .

Copied to clipboard

Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, Jing Zhang

Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025.naacl-long)

Challenge:	Speculative decoding (SD) methods are inefficient and rely on single retrieval resources.
Approach:	They propose a retrieval-based speculative decoding method that adapts the suffix automaton for efficient draft generation by utilizing the generating text sequence and static text corpus.
Outcome:	The proposed method can find the longest suffix match and can be integrated with existing methods to generalize to broader domains.

Copied to clipboard

Jacob K Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto

Text Generation

Challenge:	Existing methods to accelerate large language model inference are limited by the reliance on incremental token generation in existing draft models.
Approach:	They propose an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences and allows parallelization of both the drafting and verification steps.
Outcome:	The proposed approach provides 7.2x speedups over standard generation processes and 1.75x speed ups over existing speculative decoding approaches.

MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (2025.findings-emnlp)

Copied to clipboard

Mugilan Ganesan, Shane Segal, Ankur Aggarwal, Nish Sinnadurai, Sean Lie, Vithursan Thangarasa

PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding (2025.findings-acl)

Challenge:	Speculative decoding of vision-language models provides a novel way to accelerate language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously.
Approach:	They propose a technique that allows a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously.
Outcome:	The proposed technique increases accepted length by 30% and delivers speedups of up to 1.46x compared to conventional text-only drafting baselines on visually-grounded tasks.

Copied to clipboard

Bradley McDanel, Sai Qian Zhang, Yunhai Hu, Zining Liu

Challenge:	Speculative decoding is limited by sequential stage dependencies that prevent full hardware utilization.
Approach:	They propose a framework that generalizes speculative decoding to use multiple models arranged in a hierarchical pipeline and enables asynchronous execution with lightweight coordination for prediction verification and rollback.
Outcome:	The proposed framework achieves 2.25 tokens/unit through pipelined parallelism with multiple models arranged in a hierarchical pipeline.

Graph-Structured Speculative Decoding (2024.findings-acl)

Copied to clipboard

Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan

SPECTRA: Faster Large Language Model Inference with Optimized Internal and External Speculation (2025.acl-long)

Challenge:	Speculative decoding is a promising technique to accelerate the inference of Large Language Models.
Approach:	They propose a method that uses a token graph to record multiple sequence hypotheses within a single draft stage.
Outcome:	The proposed method significantly accelerates the inference of Large Language Models (LLMs) it allows the LLM to choose from and select the longest sequence that meets its standards.

Copied to clipboard

Nguyen-Khang Le, Truong Dinh Do, Le-Minh Nguyen

Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding (2024.emnlp-main)

Challenge:	Existing approaches to inference with Large Language Models (LLMs) are expensive and time-consuming.
Approach:	They propose a framework for accelerating large language model inference without additional training or modification to the original LLM.
Outcome:	The proposed framework outperforms state-of-the-art methods and achieves 4.08x speedups across benchmarks and LLM architectures.

Copied to clipboard

Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu, Chaojun Xiao, Xinrong Zhang, Yewei Fang, Kaihuo Zhang, Zhiyuan Liu, Maosong Sun

Challenge:	Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) drafting efficiency has become a bottleneck in the final speedup of speculative drafting, therefore generating longer drafts at less cost can lead to better speedup.
Approach:	They propose a method that uses existing model to drafting and target LLM to verify draft in a low-cost parallel manner.
Outcome:	The proposed method can achieve speedups of up to 2.4 over speculative decoding and 3.9 over vanilla decoding without fine-tuning draft and target models.

Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation (2024.findings-emnlp)

Copied to clipboard

Siru Ouyang, Shuohang Wang, Minhao Jiang, Ming Zhong, Donghan Yu, Jiawei Han, Yelong Shen

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration (2026.findings-acl)

Challenge:	Speculative decoding is a novel method to expedite inference in autoregressive (large) language models.
Approach:	They propose to use a smaller model as a draft model to speculate a block of tokens, which the target model then evaluates for acceptance.
Outcome:	The proposed method can be used to accelerate inference in autoregressive (large) language models by using smaller models as draft models to speculate tokens for multiple inference steps.

Copied to clipboard

Zhuofan Wen, Yang Feng

From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning (2026.findings-acl)

Challenge:	Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models.
Approach:	They propose a self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty.
Outcome:	The proposed framework suppresses spurious confidence and bounds speculation length based on token-wise decoding difficulty.

Copied to clipboard

Kiran Purohit, Ramasuri Narayanam, Soumyabrata Pal

Responsible & Trustworthy NLP

Challenge:	Speculative decoding (SD) allows a lightweight draft model to propose outputs that a stronger target model verifies.
Approach:	They propose a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals.
Outcome:	Experiments show that SpecGuard outperforms both SD and reward-guided SD in accuracy and reliability tests.

Speculative Sampling via Exponential Races (2025.findings-acl)

Copied to clipboard

Szymon Kobus, Deniz Gunduz

Challenge:	Speculative decoding accelerates large language model inference using a smaller draft model.
Approach:	They propose a speculative decoding method that generates multiple draft tokens for each model evaluation using a more efficient draft model.
Outcome:	The proposed method matches state-of-the-art performance and is based on exponential races.

Speculative Streaming: Efficient and Scalable Speculative Decoding with Multi-Stream Attention (2025.emnlp-main)

Copied to clipboard

Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Antonie Lin, Mohammad Rastegari, Mahyar Najibi

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding (2026.findings-acl)

Challenge:	Speculative decoding is a prominent technique for accelerating LLM inference by leveraging an auxiliary draft model, but its effectiveness is limited by the autoregressive nature of draft generation.
Approach:	They propose a method that integrates speculative draft generation directly within the target model using multi-stream attention.
Outcome:	The proposed method improves acceptance but also latency and speculation latency, limiting overall speedup.

Copied to clipboard

Zihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang, Hai Zhao

DReSD: Dense Retrieval for Speculative Decoding (2025.findings-acl)

Challenge:	Existing methods for decoding large language models generate one token per step, causing high inference latency.
Approach:	They propose a method that integrates retrieved exact patterns with logit-driven future cues.
Outcome:	Experiments on Spec-Bench, HumanEval, and MGSM-ZH show that RACER outperforms training-free methods and accelerates inference.

Copied to clipboard

Milan Gritta, Huiyin Xue, Gerasimos Lampouras

DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding (2026.findings-acl)

Challenge:	Speculative decoding (SD) uses an efficient draft model to propose the next few tokens, which are verified by the LLM in a single forward call, reducing latency while preserving its outputs.
Approach:	They propose a draft model that proposes the next few tokens from a non-parametric datastore and uses a framework that uses approximate nearest neighbour search with contextualised token embeddings to retrieve the most semantically relevant sequences for SD.
Outcome:	The proposed framework achieves (on average) 87% higher acceptance rates, 65% longer accepted tokens and 19% faster generation speeds compared to sparse retrieval (REST).

Copied to clipboard

Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, Jun Wang

Training-Free Adaptive Speculative Decoding via Linguistic Priors (2026.findings-acl)

Challenge:	Autoregressive (AR) decoding in large language models is latency-bounded by strictly sequential token generation.
Approach:	They propose a diffusion-based drafter that proposes multi-token candidates and then verifies them in parallel by the target model.
Outcome:	The proposed drafter generates multi-token proposals in a single forward pass while remaining compatible with standard AR verifiers.

Copied to clipboard

Jingyi Wang, Jiaqi Huang, Zunnan Xu, Jun Zhou, Kehong Yuan, Xiang Qian

Syntactic Text Processing

Challenge:	Speculative decoding (SPD) is a promising technique to accelerate Large Language Models (LLMs). current approaches neglect the inherent heterogeneity of natural language and fail to distinguish between semantically-rich content and structurally-predictable syntax.
Approach:	They propose a training-free framework that leverages linguistic priors to enable adaptive drafting and verification.
Outcome:	The proposed framework significantly accelerates inference without additional training.

Mamba Drafters for Speculative Decoding (2025.findings-emnlp)

Copied to clipboard

Daewon Choi, Seunghyuk Oh, Saket Dingliwal, Jihoon Tack, Kyuyoung Kim, Woomin Song, Seojin Kim, Insu Han, Jinwoo Shin, Aram Galstyan, Shubham Katiyar, Sravan Babu Bodapati

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference (2026.acl-long)

Challenge:	Existing drafters that use external drafters suffer from slower drafting while self-speculation methods use drafters tailored to the target model but require re-training.
Approach:	They propose a drafter based on a state space model, Mamba, as a solution that combines the best aspects of both approaches.
Outcome:	The proposed drafters outperform existing drafters while using less memory and maintaining their cross-model adaptability.

Copied to clipboard

null Zhouxuwen, Fangxin Liu, Chao Wang, Xiao Zheng, Hao Zheng, Min He, Li Jiang, Haibing Guan

CLaSp: In-Context Layer Skip for Self-Speculative Decoding (2025.acl-long)

Challenge:	Speculative decoding (SD) is a powerful and efficient way to accelerate autoregressive generation.
Approach:	They propose a training-free framework that recovers valid tokens discarded by standard verification . they use online correction memory and Semantic Consistency Gating to analyze rejections .
Outcome:	The proposed framework outperforms existing methods and achieves peak throughput speedup of 2.33x.

Copied to clipboard

Longze Chen, Renke Shan, Huiming Wang, Lu Wang, Ziqiang Liu, Run Luo, Jiawei Wang, Hamid Alinejad-Rokny, Min Yang

Challenge:	Existing methods for drafting Large Language Models require additional modules to be trained, which can be challenging to implement and ensure compatibility across various LLMs.
Approach:	They propose an in-context layer-skipping strategy for self-speculative decoding that uses a plug-and-play mechanism to skip intermediate layers of the verify model to construct a compressed draft model.
Outcome:	The proposed method achieves a speedup of 1.3 1.7 on LLaMA3 series models without altering the original distribution of the generated text.

Multi-Drafter Speculative Decoding with Alignment Feedback (2026.findings-acl)

Copied to clipboard

Taehyeon Kim, Hojung Jung, Se-Young Yun

Challenge:	Existing methods to accelerate large language model (LLM) inference use a smaller model to draft future tokens, which are then verified by the target LLM.
Approach:	They propose a unified framework that integrates multiple drafters into the SD process.
Outcome:	Extensive experiments show that MetaSD outperforms single-drafter approaches.

LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation (2026.findings-acl)

Copied to clipboard

Tianyu Liu, Qitan Lv, Hao Li, Xing Gao, Xiao Sun, Xiaoyan Sun

Speculative Verification: Exploiting Information Gain for Speculative Decoding (2026.findings-acl)

Challenge:	Speculative decoding (SD) is a promising technique for LLM inference acceleration.
Approach:	They propose a method to generate draft tokens in a retrieval-based manner to reduce drafting overhead and improve inference speed.
Outcome:	Extensive tests show that LogitSpec can achieve 2.61 speedup and 3.28 mean accepted tokens per decoding step.

Copied to clipboard

Sungkyun Kim, Jaemin Kim, Dogyeong Yun, Jiho Shin, Junyeol Lee, Jiwon Seo

TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs (2026.acl-long)

Challenge:	Large Language Models (LLMs) are used for many applications but their size and computational cost make inference serving a significant challenge.
Approach:	They propose an efficient augmentation to Speculative Decoding (SD) that predicts speculation accuracy and dynamically adapts the verification length to maximize throughput.
Outcome:	The proposed model reduces wasted verification on rejected tokens and improves decoding efficiency.

Copied to clipboard

Sibo Xiao, Fu Jinyuan, Zhongle Xie, Lidan Shou

Challenge:	Speculative decoding (SD) is a useful tool for accelerating large language models . but its utility is limited by a fundamental constraint: draft and target models must share the same vocabulary .
Approach:	They propose an algorithm that uses a draft token sequence to get a new target token sequence and then uses DTW to build a mapping to transfer probability distributions.
Outcome:	The proposed method shows 1.57x speedup on various tasks.

Speculative Decoding with a Speculative Vocabulary (2026.findings-acl)

Copied to clipboard

Miles Williams, Young D. Kwon, Rui Li, Alexandros Kouris, Stylianos I. Venieris

EDSD: Entropy-Driven Design for Faster Speculative Decoding (2026.acl-long)

Challenge:	Speculative decoding methods use a draft model to accelerate inference while yielding identical outputs.
Approach:	They propose a method that selects a vocabulary subset per decoding step and uses a draft model to generate a series of tokens that are verified in parallel.
Outcome:	The proposed method achieves higher acceptance length than state-of-the-art speculative decoding method, EAGLE-3.

Copied to clipboard

Longkai Cheng, Ximing Wang, Jiangcai Zhu, Kailai Shao, Chao Chen, Haixiang Hu

Responsible & Trustworthy NLP

Challenge:	Existing methods for speculative decoding incur substantial training overhead to mitigate information misalignment between autoregressive draft model training and decoding.
Approach:	They propose an Entropy-Driven Speculative Decoding framework that uses entropy as a unified, interpretable signal for both draft model training and architectural design.
Outcome:	Experiments on seven large language models show that EDSD improves training efficiency by 24.8% and increases acceptance length by 4.0% compared to state-of-the-art methods.

SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences (2026.findings-acl)

Copied to clipboard

Jungyoub Cha, Hyunjong Kim, Sungzoon Cho

Challenge:	Speculative decoding performance degrades as input length increases, with significant drops even at moderate lengths.
Approach:	They propose a drop-in enhancement that improves speculative decoding on long sequences without additional training.
Outcome:	The proposed enhancement accelerates speculative decoding by up to 2.84 on 16K-token long document summarization and up to 3.86 on long-form reasoning while preserving the short-input performance of state-of-the-art frameworks.

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation (2026.acl-long)

Copied to clipboard

Zining Liu, Yunhai Hu, Tianhua Xia, BO Bao, Eric Sather, Vithursan Thangarasa, Sai Qian Zhang